Monado Android Sensor Data Flow

发表于   |   分类于   |   1,762 字   |   8 分钟   |  

Monado Android Sensor Data Flow

Flow流程

流程上，是两个线程：

  • 线程一是app的render线程，每次render的时候都需要去获取xrLocateViews，从而获取到底层Sensor的写完的数据，这里是消费者comsumer。
  • 线程二是android的sensor callback，这里会不断做数据填充的动作，这里是生产者provider。

整体的函数调用全地图如下：

img

代码解析

生产者：Sensor Callback

provider的部分比较直观，数据是通过ASensorManager_getDefaultSensor(d->sensor_manager,GSENSOR_TYPE_GYROSCOPE);来设置callback的。因此整个的流程也就是从这里触发的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
android_sensors.c，libopenxr_monado.so
static int
android_sensor_callback(int fd, int events, void *data)
{
	struct android_device *d = (struct android_device *)data;
	......
	while (ASensorEventQueue_getEvents(d->event_queue, &event, 1) > 0) {
		switch (event.type) {
		......
		case ASENSOR_TYPE_GYROSCOPE: {
			gyro.x = -event.data[1];
			gyro.y = event.data[0];
			gyro.z = event.data[2];
			ANDROID_TRACE(d, "gyro %ld %.2f %.2f %.2f", event.timestamp, gyro.x, gyro.y, gyro.z);
			// TODO: Make filter handle accelerometer
			struct xrt_vec3 null_accel;
			// Lock last and the fusion.
			os_mutex_lock(&d->lock);
			m_imu_3dof_update(&d->fusion, event.timestamp, &null_accel, &gyro);
			// Now done.
			os_mutex_unlock(&d->lock);
		}
		default: ANDROID_TRACE(d, "Unhandled event type %d", event.type);
		}
	}
	return 1;
}

sensor callback的部分其实并没有太多可以说的，唯一需要注意的是在设置时有用到ASensor和GSensor，但是实际在数据处理上只是用到了GSensor的数据。

数据的处理中使用的3dof的算法（这里可以挖一个坑，需要去看一下3dof算法大概是怎么回事）。

3dof数据处理：m_imu_3dof_update

这部分的代码是在xrt/auxilary/math目录下的，因此在模块结构上属于是可替换的部分，简单来说未来是可以做为一个计算单元模块放到dsp中去的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
m_imu_3dof.c
void
m_imu_3dof_update(struct m_imu_3dof *f,
                  uint64_t timestamp_ns,
                  const struct xrt_vec3 *accel,
                  const struct xrt_vec3 *gyro)
{
	......
	if (gyro_biased_length > 0.0001f) {
		struct xrt_vec3 rot_axis = {
		    gyro_biased.x / gyro_biased_length,
		    gyro_biased.y / gyro_biased_length,
		    gyro_biased.z / gyro_biased_length,
		};
		float rot_angle = gyro_biased_length * (float)dt;
		struct xrt_quat delta_orient;
		math_quat_from_angle_vector(rot_angle, &rot_axis, &delta_orient);
		math_quat_rotate(&f->rot, &delta_orient, &f->rot);
	}
	// Gravity correction.
	gravity_correction(f, timestamp_ns, accel, &gyro_biased, dt, gyro_biased_length);
	// Gyro bias calculations.
	gyro_biasing(f, timestamp_ns);
	/*
	 * Mitigate drift due to floating point
	 * inprecision with quat multiplication.
	 */
	math_quat_normalize(&f->rot);
}

这里用到了一个数据结构：m_imu_3dof，调用的函数栈：m_imu_3dof_update(&d->fusion, event.timestamp, &null_accel, &gyro);

其中d的数据结构是：android_device，数据结构的部分如下图：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
/*!
 * @implements xrt_device
 */
struct android_device
{
	......
	struct
	{
		//! Lock for last and fusion.
		struct os_mutex lock;
		struct m_imu_3dof fusion;
	};
	......
};
struct m_imu_3dof
{
	struct xrt_quat rot; //!< Orientation
	......
};
struct xrt_quat
{
	float x;
	float y;
	float z;
	float w;
};

最终sensor callback中会填充：m_imu_3dof中的rot。

消费者：Render Thread

app端的行为有点出乎我的预料，我本来以为渲染的流程应该是在openxr的防守范围内，但是从实际的代码上来看，是在app内部，也就是说至少在提供Engine SDK的时候需要把渲染的流程也写进去，那么在SDK的层面上，至少需要保证：

  • 渲染的流程
  • 事件polling的流程

除此之外可能还需要做一些额外的事情，比如在高通现在的SXR平台上的ATW，那么同样的迁移过来：

  • TW/SW的流程

事情就变得一下子有点多了，无论如何先从这边的流程分析一下sensor callback的事件是怎么消费的。

整个调用xrLocateViews的流程其实并没有什么特别的，这一块是在先前的文章中也有介绍，在实现上是通过dlopen的方式加载so库到应用进程，从而完成了runtime加载。

重温一下定义：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
XRAPI_ATTR XrResult XRAPI_CALL xrLocateViews(
    XrSession                                   session,
    const XrViewLocateInfo*                     viewLocateInfo,
    XrViewState*                                viewState,
    uint32_t                                    viewCapacityInput,
    uint32_t*                                   viewCountOutput,
    XrView*                                     views);
#define ENTRY(funcName)                                                                                                \
	do {                                                                                                           \
		if (strcmp(name, #funcName) == 0) {                                                                    \
			PFN_##funcName ret = &oxr_##funcName;                                                          \
			*out_function = (PFN_xrVoidFunction)(ret);                                                     \
			return XR_SUCCESS;                                                                             \
		}                                                                                                      \
	} while (false)
ENTRY(xrLocateViews);

展开后：

1
2
3
4
5
6
7
8
// expend ENTRY(xrLocateViews);
do {                                                                                                           
    if (strcmp(name, xrLocateViews) == 0) {                                                                    
        PFN_xrLocateViews ret = &oxr_xrLocateViews;                                                          
        *out_function = (PFN_xrVoidFunction)(ret);                                                     
        return XR_SUCCESS;                                                                             
    }                                                                                                      
} while (false)

因此通过函数跳转链：xrLocateViews -> oxr_xrLocateViews直接定位到：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
XrResult
oxr_xrLocateViews(XrSession session,
                  const XrViewLocateInfo *viewLocateInfo,
                  XrViewState *viewState,
                  uint32_t viewCapacityInput,
                  uint32_t *viewCountOutput,
                  XrView *views)
{
	OXR_TRACE_MARKER();
	......	
	return oxr_session_locate_views( //
	    &log,                        //
	    sess,                        //
	    viewLocateInfo,              //
	    viewState,                   //
	    viewCapacityInput,           //
	    viewCountOutput,             //
	    views);                      //
}
XrResult
oxr_session_locate_views(struct oxr_logger *log,
                         struct oxr_session *sess,
                         const XrViewLocateInfo *viewLocateInfo,
                         XrViewState *viewState,
                         uint32_t viewCapacityInput,
                         uint32_t *viewCountOutput,
                         XrView *views)
{
	struct xrt_device *xdev = GET_XDEV_BY_ROLE(sess->sys, head);
	......
	if (sess->sys->inst->debug_views) {
		U_LOG_D("viewLocateInfo->displayTime %" PRIu64, viewLocateInfo->displayTime);
	}
	// Get the viewLocateInfo->space to view space relation.
	struct xrt_space_relation pure_relation;
	oxr_space_ref_relation(           //
	    log,                          //
	    sess,                         //
	    XR_REFERENCE_SPACE_TYPE_VIEW, //
	    baseSpc->type,                //
	    viewLocateInfo->displayTime,  //
	    &pure_relation);              //
	......
	for (uint32_t i = 0; i < view_count; i++) {
		struct xrt_pose view_pose = XRT_POSE_IDENTITY;
		// Get the per view pose from the device.
		xrt_device_get_view_pose(xdev, &eye_relation, i, &view_pose);
		// Do the magical space relation dance here.
		struct xrt_space_relation result = {0};
		struct xrt_relation_chain xrc = {0};
		m_relation_chain_push_pose_if_not_identity(&xrc, &view_pose);
		m_relation_chain_push_relation(&xrc, &pure_relation);
		m_relation_chain_push_pose_if_not_identity(&xrc, &baseSpc->pose);
		m_relation_chain_resolve(&xrc, &result);
		......
	}
	return oxr_session_success_result(sess);
}

这一段的调用函数中其实分为两部分：

  • oxr_space_ref_relation
  • xrt_device_get_view_pose

oxr_space_ref_relation

其中oxr_space_ref_relation的部分是概况图中间的部分，最终会调用：android_device_get_tracked_pose，可以看到其实就是获取了：android_device.m_imu_3dof.rot

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
android_sensor.c
static void
android_device_get_tracked_pose(struct xrt_device *xdev,
                                enum xrt_input_name name,
                                uint64_t at_timestamp_ns,
                                struct xrt_space_relation *out_relation)
{
	(void)at_timestamp_ns;
	struct android_device *d = android_device(xdev);
	out_relation->pose.orientation = d->fusion.rot;
	//! @todo assuming that orientation is actually currently tracked.
	out_relation->relation_flags = (enum xrt_space_relation_flags)(XRT_SPACE_RELATION_ORIENTATION_VALID_BIT |
	                                                               XRT_SPACE_RELATION_ORIENTATION_TRACKED_BIT);
}

  • 这边值得注意的是在中间的函数调用链上：

    • oxr_session.c，oxr_session_locate_views
    • oxr_space.c，oxr_space_ref_relation
      • 这里space == XR_REFERENCE_SPACE_TYPE_VIEW
    • oxr_session.c，oxr_session_get_view_relation_at
      • oxr_xdev_get_relation_chain(log, sess->sys->inst, xdev, XRT_INPUT_GENERIC_HEAD_POSE, at_time, &xrc);
      • 这里的参数中：XRT_INPUT_GENERIC_HEAD_POSE，实际在最后并没有用到，因此在新的平台上要注意一下
    • oxr_xdev.c，oxr_xdev_get_relation_chain
    • xrt_device.h，xrt_device_get_tracked_pose
    • android_sensor.c，android_device_get_tracked_pose
      sequenceDiagram
oxr_session.c ->> oxr_space.c : oxr_session_locate_views
oxr_space.c ->> oxr_session.c : oxr_space_ref_relation
oxr_session.c ->> oxr_xdev.c : oxr_session_get_view_relation_at
oxr_xdev.c ->> xrt_device.h : oxr_xdev_get_relation_chain
xrt_device.h ->> android_sensor.c : xrt_device_get_tracked_pose

整个数据的调用链还是比较清晰的，最后的输出参数为：xrt_space_relation pure_relation，其中的pose为

1
2
3
4
5
6
7
8
9
10
11
12
13
struct xrt_space_relation
{
	enum xrt_space_relation_flags relation_flags;
	struct xrt_pose pose;
	struct xrt_vec3 linear_velocity;
	struct xrt_vec3 angular_velocity;
};
struct xrt_pose
{
	struct xrt_quat orientation;
	struct xrt_vec3 position;
};

其中xrt_space_relation.xrt_pose.position的部分并没有被赋值，而xrt_space_relation.xrt_pose.orientation为：

1
out_relation->pose.orientation = d->fusion.rot;

xrt_device_get_view_pose

view pose最后是调用到u_device.c的u_device_get_view_pose，这一段函数看起来只是在做一般的偏移纠正，并没有看太明白。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
void
u_device_get_view_pose(const struct xrt_vec3 *eye_relation, uint32_t view_index, struct xrt_pose *out_pose)
{
	struct xrt_pose pose = XRT_POSE_IDENTITY;
	bool adjust = view_index == 0;
	pose.position.x = eye_relation->x / 2.0f;
	pose.position.y = eye_relation->y / 2.0f;
	pose.position.z = eye_relation->z / 2.0f;
	// Adjust for left/right while also making sure there aren't any -0.f.
	if (pose.position.x > 0.0f && adjust) {
		pose.position.x = -pose.position.x;
	}
	if (pose.position.y > 0.0f && adjust) {
		pose.position.y = -pose.position.y;
	}
	if (pose.position.z > 0.0f && adjust) {
		pose.position.z = -pose.position.z;
	}
	*out_pose = pose;
}

当获取到view pose后，以及上面得到的relation pose orientation，会参与到

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
oxr_session.c , oxr_session_locate_views	
XrResult
oxr_session_locate_views(struct oxr_logger *log,
                         struct oxr_session *sess,
                         const XrViewLocateInfo *viewLocateInfo,
                         XrViewState *viewState,
                         uint32_t viewCapacityInput,
                         uint32_t *viewCountOutput,
                         XrView *views)
{
    ......
	for (uint32_t i = 0; i < view_count; i++) {
		struct xrt_pose view_pose = XRT_POSE_IDENTITY;
		// Get the per view pose from the device.
		xrt_device_get_view_pose(xdev, &eye_relation, i, &view_pose);
		// Do the magical space relation dance here.
		struct xrt_space_relation result = {0};
		struct xrt_relation_chain xrc = {0};
		m_relation_chain_push_pose_if_not_identity(&xrc, &view_pose);
		m_relation_chain_push_relation(&xrc, &pure_relation);
		m_relation_chain_push_pose_if_not_identity(&xrc, &baseSpc->pose);
		m_relation_chain_resolve(&xrc, &result);
		union {
			struct xrt_pose xrt;
			struct XrPosef oxr;
		} safe_copy_pose = {0};
		safe_copy_pose.xrt = result.pose;
		views[i].pose = safe_copy_pose.oxr;
		// Copy the fov information directly from the device.
		union {
			struct xrt_fov xrt;
			XrFovf oxr;
		} safe_copy_fov = {0};
		safe_copy_fov.xrt = xdev->hmd->views[i].fov;
		views[i].fov = safe_copy_fov.oxr;
		......
	}
    ......
}

整个计算过程（现在还没有搞明白），

1
2
3
4
m_relation_chain_push_pose_if_not_identity(&xrc, &view_pose);
m_relation_chain_push_relation(&xrc, &pure_relation);
m_relation_chain_push_pose_if_not_identity(&xrc, &baseSpc->pose);
m_relation_chain_resolve(&xrc, &result);

所以当运行到这里的时候，从sensor那边获取的GSensor data就被用完了。

总结

  • 数据是使用Android的Sensor Callback填充而来，封装成：android_device.m_imu_3dof.xrt_quat
    • 速率：#define POLL_RATE_USEC (1000L / 60) * 1000，16.6ms，1秒60次
  • 消费者是App的xrLocateView其中oxr_space_ref_relation，每次会去读取最新的android_device.m_imu_3dof.xrt_quat
    • 速率由App端决定

整个流程是一个粗略的认知，但是至少明白了事件往app的送的流程，还存在不少的坑，后面需要厘清的部分有：

  • [ ] 3dof 算法
  • [ ] m_relation_chain的用法/意义
  • [ ] 除了XRT_INPUT_GENERIC_HEAD_POSE，其他的Action是怎么封装的